Independent Variable Group Analysis in Learning Compact Representations for Data

Humans tend to group together related properties in order to understand complex phenomena. When modeling large problems with limited representational resources, it is important to be able to construct compact models of the data. Structuring the problem into sub-problems that can be modeled independently is a means for achieving compactness. We describe the Independent Variable Group Analysis (IVGA), an unsupervised learning principle that in modeling a data set, also discovers a grouping of the input variables that reflects statistical independencies in the data. In addition, we discuss its connection to some aspects of cognitive modeling and of representations in the brain. The IVGA approach and its implementation are designed to be practical, efficient, and useful for real world applications. Initial experiments on several data sets are reported to examine the performance and potential uses of the method. The preliminary results are promising: the method does seem to find independent subsets of variables. Moreover, it leads to markedly more compact and efficient models than the full model without variable grouping. This allows the re-allocation of freed representational resources for other important tasks. Compact models also contain much fewer parameters and generalize better, and therefore require less data for estimation.